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ABSTRACT 

This booklet is second in a series of guides for 
formative evaluation of education programs. The guide represents a 
joint project to develop and implement a system for ongoing formative 
and summative evaluation of the Department of Energy's (DOE) programs 
to improve precollege mathematics and science education. The guide 
focuses on the use of the "Template for Teacher Development 
Programs, M which allows for the description of a program and for 
comparison of that program to best practice. The template identifies 
what research and experience have been found to work, and is the key 
tool in the first phase of evaluation of DOE precollege education 
programs, called "program profiling." Section I covers the 
development of templates, anatomy of a template, guidelines for use 
of the template, summarizing the results of the profiling process, 
and sample questions for profiling teacher development programs. 
Section II contains the Template for Teacher Development Programs and 
descriptive and context information. Section III provides operational 
definitions to clarify the following components and subcomponents of 
the Template: program administration, vision for the classroom, 
teacher development program activities, follow-up, teacher 
leadership, program evaluation, and descriptive and context 
information. Section IV contains 19 references to research. (LZ) 
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Introduction 



i 

■ 

This booklet is the second in a series of guides for formative evaluation of education 

■ programs developed by the National Center for Improving Science Education in 
collaboration with the U.S. Department of Energy (DOE). The work described herein is 
one component of a four-year project. Funded by DOE, this project is a joint endeavor 

■ to develop and implement a system for ongoing formative and summative evaluation of 
DOE's programs to improve precollege mathematics and science education. 

■ In the past several years the DOE has initiated a number of precollege programs serving 
teachers and students across the country. In 1990, for example, over 200,000 teachers 
and students participated in DOE-funded activities. The programs draw heavily on 
E DOE's unique capabilities: outstanding scientists, state-of-the-art lab facilities, and 

m research that crosses into frontier science. The precollege teacher programs have two 

■ goals: (1) to enhance teachers' knowledge in science and the research process and (2) to 
strengthen their teaching of science and mathematics in order to enhance student 
outcomes, including achievement and persistence in pursuing scientific and technical 
M fields. 

DOE's large-scale investment in precollege education demands accountability. DOE 
m must know what is actually happening in the programs, what impact the programs are 

| having, and what needs to be done to improve, stabilize, and institutionalize them to 

contribute to achieving the National Educational Goals. 

I This booklet is a guide to the process of "program profiling," an approach to formative 

evaluation used by the National Center for Improving Science Education and developed 

■ originally by its host organization, The NETWORK, Inc. The type of program addressed 
in this guide is teacher development. Similar guides for teacher research participation 
programs, student programs, and systemic programs are being developed. Although 
these guides are based on DOE programs, the assessment process and the accompanying 
instrumentation can readily be applied to similar programs sponsored or conducted by 
other federal agencies, businesses and industries, nonfederal organizations, and schools 
or school districts. 

The developer of. this booklet, the National Center for Improving Science Education, 
promotes change in state and local policies and practices in science curriculum, teaching, 
and assessment. In carrying out its work, the Center provides a range of products and 
services to education policy makers and practitioners who work to strengthen science 
teaching and learning across the country. In addition to its work in program evaluation, 
the Center reports and makes recommendatioas on major trends and reform efforts m 
science education, is documenting key innovation efforts across the country, and is an 
active participant in the development and analysis of international studies on curricula, 
educational practices, and student achievement in science, mathematics, and technology. 
The Center also provides technical assistance at all levels of the education system, 
ranging from implementation of formative evaluation to training teachers m hands-on 
science to improve science education in a district or state. 
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SECTION ONE 



A TOOL FOR PROFILING 
TEACHER DEVELOPMENT PROGRAMS 
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Purpose 



Significant investments of time, energy) and resources are currently being made in a wide 
variety of programs that aim to improve education. The Department of Energy is one 
such investor, with the mission of improving scudent learning in mathematics, science, 
and engineering technology through the enhancement of teacher knowledge and skills. A 
wide variety of programs are currently being funded by DOE that target precollege 
education, providing unique learning opportunities for both students and teachers. But 
how good are these programs? Is the funding being invested wisely? What adjustments 
could be made to better use the available resources and, as resources shrink, where 
should the cuts be made? 

In order to address these important i 5 ^act questions, a prior question must be answered: 
What is actually happening in the programs? Without the answer ;o this question, it is 
impossible to know what is causing either positive or negative learning outcomes. 
Likewise, it is impossible to know what to change for the purposes of improvement. The 
Template for Teacher Development Programs was developed as a tool to help answer this 
question. 

The template has an additional purpose as well. Instead of allowing only for the 
description of a program, it also allows for comparison of that program to best practice. 
The template identifies what research and experience have found to work - elements 
that, when part of a program's design, should enhance its effectiveness. Therefore, the 
template can also help answer another important question about the program: To what 
extent does it reflect best practice? Addressing this question helps program developers 
and managers identify components of their programs that could be the focus of 
improvement. 

The template is thus a formative evaluation tool, as well as one of several instruments 
needed to evaluate the impact of a program. It is the key tool used in the first phase of 
evaluation of DOE precollege education programs - called Program Profiling - and one 
of several data collection tools used in the second phase - Impact Assessment. 



Development of Templates 

The templates for Department of Energy programs have been developed collaboratively, 
with the National Center for Improving Science Education playing a lead role, and with 
full participation from precollege education program managers and staff from DOE 
national laboratories and other facilities. Early in the collaboration, it was determined 
that there was such a variety of programs that more than one template would be 
necessary to capture their important features. Several program types were identified in 
the first discussion: tes her research participation programs, teacher development 
programs, student programs, systemic programs, and special programs. It was decided 
that over the course of three years, a template would be developed for each. 

Template development begins with searching and reading the literature on each program 
type, focusing particularly on what research and practice indicate to be components 
necessary for such programs to be effective. During meetings of DOE and DOE lab 
staff, readings (mainly syntheses) selected by the Center were discussed and components 
of effective practice identified through small group work. Center staff combined the 
work of groups into a draft template and circulated it for review. 

It is important to note that some of the program types have not been studied carefully to 
date, and so some components on the templates are therefore based largely on the 
experience of program staff. There is a large literature and research base for Teacher 
Development Programs in general; although limited research is available for Teacher 
Development in research laboratories conducted by scientific personnel. This is viewed 
as an opportunity to add to the knowledge base on effective practice; the components 
listed are being tested for their effectiveness during the evaluation effort. 

Once drafted, each template was piloted in several labs. Five labs where Teacher 
Development Programs were being conducted were selected for the pilot study using a 
set of criteria developed collaboratively. Center staff read descriptive material about 
each lab's Teacher Development Program and made site visits during which program 
staff, teachers, and other key players were interviewed and observed. A template was 
filled out profiling each lab's program; at the same time, refinements were made in the 
template. (A Pilot Profiling Report is available from the Center.) 

Based on learnings from the pilot of how best to use the template to profile programs, 
the final draft of the template was then used by all labs to profile their own Teacher 
Development Programs. A summary of the profiles is being compiled by the Center and 
will be available by mid- 1994. 

This same process will be used to develop, pilot, and revise the template for each 
program type. Following the profiling of each type of DOE program, designs and 
instruments for assessing impact will be developed and tested by the Center in the 
second phase of this collaborative project. Once the results of the pilot impact 
assessments are in, the template will be reviewed once more and revised a final time - 
this time with a better understanding of the components that make such programs 
effective. 



2 10 



Anatomy of a Template 



The template in Section 2 provides a format for profiling a program as it reflects best 
practice. It is formatted in three columns. The first column lists Components of 
Effective Practice, as determined by the template's developers from research and 
experience. The components are concisely worded, as appropriate to this kind of a tool, 
and some may find they need clarification and/or elaboration. For this purpose, 
operational definitions are included (see Section 3) for those terms deemed ambiguous, 
technical, or uncommon; a reference list (see Section 4) is also included, with the 
components on the template footnoted as to which reference provides the supporting 
information. 

The second column provides the opportunity to describe the program as it is designed to 
work, i.e., the Intended Program. The third column, then, is used to describe what 
happens when the Actual Program is functioning. 

Information from these columns can be used to address the following questions: 

1. What ig best practice for these types of programs? 

2. To what extent is the program designed to reflect what is most effective? 

3. To what extent does the program in place reflect best practice? 

4. To what extent is the program's design actually carried out? 

5. Where are the gaps? What can be improved? 

The last page of the template provides the opportunity to describe addition* 
characteristics of the program and of its participants chat may have an impact on its 
effectiveness. For example, certain program characteristics may work differently with 
different teachers (e.g., elementary vs. secondary, experienced vs. novice, teachers of 
advanced classes vs. general classes), or with different Department of Energy lab 
characteristics (e.g., hands-on vs. theoretical, high technology-based vs. limited 
technology). 

Particularly important for DOE programs is the description of the unique competencies, 
facilities, personnel, and other attributes that each program offers to its participants. 
This is critical to DOE's acceptance, and, indeed, welcome, as an important part in the 
education of young people in the areas of science, mathematics, and engineering 
technology* The competencies and capabilities listed here can then be cross-referenced 
with other parts of the template where they are mentioned. For example, for Teacher 
Development Programs, the question can be asked: Are science teachers learning in the 
program something that relates to the "frontier science" being conducted at the lab? 




Guidelines for Use of the Template 



There are many ways that profiling can be done, depending on the situation. The more 
people involved, the more this activity can resemble a staff development activity. For 
example, staff and participants in a program might be asked to complete the template 
and form discussion groups to share their observations, experiences, and impressions. Or 
a smaller group of those involved might do the same. 

Another option is for program managers to select and interview people; a better option 
is for program managers to form a small evaluation team representative of the different 
roles who then select and do the interviews. A final option is to have someone from 
outside the program do the profiling. 

It is important that data be gathered from people in every role (e.g., program staff, 
teacher participants, mentors, other key participants) and from more than one person in 
each role. The more people, the fuller the picture. 

There is no structured protocol for use by those doing the profiling. They must instead 
be guided by the information they know they need to acquire for completion of the 
template. Interviews, observations, and document analysis should be approached in such 
a way as to achieve this. 

Here are some guidelines for Department of Energy lab staff gathering information to 
complete the template: 

1. Review all program materials carefully. There are places in the template 
that ask for goals, for a description of materials sent to participants, for 
specific follow-up activities, etc. These materials are important pieces of 
information. 

2. A general note about interviewing: use as many open-ended questions as 
possible rather than going through the template item-by-item and asking 
specific questions. (A list of possible interview questions for teachers follows 
these guidelines, to model what it means to be open-ended.) When you 
interview you want the interviewee to do as much of the talking as possible. 

3. Interview program developers and managers, with a special emphasis on how 
they have designed the program. What key elements do they think are 
important? How have those played out? What's actually happening? What 
has been particularly successful and what less so? How have they changed 
the elements over the years and why? If they don't mention the specific 
components from the template, ask them. Be sure to get the descriptive and 
context information listed on the last page. 
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4. Interview program participants (teachers, mentors, etc.) to learn what their 
experiences have been. What seem to be the most important elements of 
the program? What hasn't worked? How does this kind of experience differ 
from other professional development programs theyVe been involved in? If 
they don't mention the specific components from the template, ask them. 
(See list of possible questions.) Interview at least 10% of the participants, 
but no fewer than eight. An alternative is to convene small focus groups of 
two or three people. In either case, select a cross-section of people, varying 
on such dimensions as kind of school, attitude about the program, etc. 

5. Observe, whenever possible, to get a sense of the environment, the people, 
the work, the interactions, the facilities, and the stories. Follow up with 
questions about your observation to enhance the quality and quantity of 
information. 

6. As you complete the template, avoid writing 'yes" or "no," but instead 
describe how the component looks or should look. For example, the item 
may read, "creates collegial atmosphere." Rather than saying "yes" under 
Intended Program, you might say "teachers are housed in clusters to promote 
interaction; once a week lunches are held for all participants and their 
mentors; Friday afternoon evaluation discussions probe how teachers feel." 
This kind of response is more helpful than "yes." 



Summarizing the Results of the Profiling Process 

Data about the program are analyzed and synthesized in a way that is most useful for 
program managers considering program improvement. At minimum, a template reflects 
the program as it is intended and as it actually occurs. Conflicting data, e.g., when 
different people's reports of a given component vary, provide useful information. The 
following questions can then be asked, in comparing columns on the template: 

• To what extent does the program design reflect best practice? 

• To what extent are program components actually implemented? 

• To what extent does what is happening in the program reflect best 
practice? 

• To what extent does the program take full advantage of the unique 
capabilities of the Department of Energy lab? 

• What changes should be considered in the program, given its goals and 
constraints? 



Sample Questions for Profiling Teacher 
Development Programs 

The purpose of these questions is to provide sufficient information to complete the 
template's Actual Program column- These are questions for teachers, but many could be 
adapted easily for mentors and program managers. 

As an opener, begin with a general question about the program, for example, 

If you were going to give a presentation about your experience, what are 
the four or five things you'd want to be sure to talk about? 

This should give you information about some of the components and some of the sub- 
components. 

Then follow with open-ended questions about each of the components. The questions 
flow better conversationally than numerically, for example, 

(Component 1) Program Administration 

What do you see as the goals of this program? 

Note: Additional questions for this component are found at the end of this section. 

(Component 2) Vision for the Classroom 

The program you are attending is based on a vision of good science education for 
kids and includes what a good science program would look like in the classroom. 
Could you describe that vision or parts of that vision for me? Probe for specifics 
such as: 

What would you be teaching? 

How would you be teaching? 

What would the kids be doing? 

What learning outcomes would you be working for? 

How would you assess what the kids had learned? 

How would your teaching accommodate the differences in your kids? 



(Component 3) Teacher Development Program Activities 

What different approaches are used by those running the program to help you 
learn? Probe for specifics such as: 

How does a typical day go? 

What do you do during a typical day? 

What opportunities "back home" do you have to practice what you are learning? 

(Component 4) Unique Contribution of DOE Laboratories 

Teacher development programs are sponsored by a wide variety of organizations. 
How is this program special (or different) because it is sponsored by the Lab? or 
What do you think the Lab brings to this program that makes it different from 
others? Probe for specifics such as: 

How do scientists contribute? 

How do the facilities contribute? 

How does the content differ? 

In what ways is the content you are learning linked to the mission of the Lab? 

(Component 5) Follow-Up; (Component 6) Teacher Leadership and Responsibility 

Tell me about your expectations for after the program. What ore the expectations 
for you after you return home? What expectations do you have for support or 
continued connections to the Lab or Lab staff? Please be specific. 

(Component 6) Teacher Leadership and Responsibility; (Component 7) Program 
Evaluation 

Have you had any responsibilities beyond those of being a learner in the program 
such as: 

Have you been asked how things are going? 

Have you noticed changes in anything that may have resulted from gathering 
evaluation information from you and other participants? 
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(Component 1) Program Administration 

How have the people in charge of this program made it work for you? Probe for 
specifics such as: 

Were you well prepared? 

Were arrangements clear? 

What land of atmosphere have they provided? 

Are you clear about any future expectations or follow-up? 
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SECTION TWO 



TEMPLATE FOR TEACHER 
DEVELOPMENT PROGRAMS 
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Operational Definitions 

H How would I know if I said it?" The "following definitions are intended to help answer 
that question and to clarify components and subcomponents of the Teacher Development 
Program template. Some components are cross-referenced. 

Component 1: Program Administration 

(d) Collegial Atmosphere: An environment in which professional exchange is 
encouraged and mechanisms are created to make it happen. For example, opportunities 
are scheduled for teachers and/or scientists to discuss research or teaching topics; 
program staff are always accessible (have an "open door" policy); participants are 
encouraged to pursue their research interests by linking with scientists; conversation 
includes talk about work, the craft of teaching or scientific investigation; people listen to 
each other, value each other's contributions, treat each other as peers. 

(j) Underrepresented: Refers to females, non-Asian minorities, and the poor. These 
groups whose established patterns of lower levels of school and adult accomplishments in 
scientific and quantitative fields put them educationally at risk in these subject areas 
(from Indicators for Monitoring Mathematics and Science Education: A Sourcebook. 
RAND Corporation, 1989). 



Component 2: Vision for the Classroom 

This component describes the kind of science programs labs want teachers to use with 
their students. 

(a) Deep Understanding: Helping students learn so that they can apply their learnings to 
new situations, as opposed to superficial learning that they can merely parrot back. 

Major science concepts or principles, development of skills, and "scientific habits of 
mind;" important outcomes of the science programs that lab staff are helping teachers to 
deliver to their students. Major science concepts or principles may be those called 
themes or organizing concepts in AAAS ! s Science for All Americans or reports from the 
National Center for Improving Science Education (NCISE) such as change, systems, or 
diversity, or they may be more specific, such as the concept of a food web or 
radioactivity. What is important is that there is not a focus on the learning of 
disconnected facts or vocabulary words. Skills include generic thinking skills (e.g., critical 
thinking, reasoning) science-related thinking or process skills (e.g., critical thinking, 
designing investigations, inferring, predicting), science lab skills (e.g., using equipment, 
measurement). "Scientific habits of mind" include attitudes and dispositions important to 
scientific thinking and investigations, such as honesty, skepticism, tolerance of ambiguity, 
desire for evidence. (See NCISE reports for more detail.) 
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(b) "Hands-on/minds-on" Instructional Approach: Labs help teachers develop an 
inquiry-based approach to teaching that not only includes hands-on experience or 
activities, but includes ways for students to develop new understandings from these 
activities. This may be through the use of a "teaching/learning model" such as the 4- 
stage one proposed by NCISE: (1) invitation to learn, (2) explore, discover, investigate, 
(3) propose explanations, (4) apply in different situations; or the ,! 5E M model of BSCS: 
(1) engage, (2) explore, (3) explain, (4) elaborate, (5) evaluate; or another similar model. 
All these models emphasize questioning, investigation, and application - with attention 
to pulling the learning together after investigation (not lecture or reading, followed by a 
lab experience for confirmation or reinforcement). They also avoid simply doing 
activities. 

(c) Depth of Study Rather Than Breadth: Labs help teachers develop programs or 
teaching strategies that explore fewer concepts in greater depth than is common practice 
in schools, where the amount of vocabulary in science texts exceeds that in foreign 
language texts. Teachers need to be selective and delete or give superficial attention to 
some concepts and skills while exploring others in depth. 

(d) Balance Between Science Content and Process: Inclusion of both knowledge and 
skills as important outcomes of school science. 

(e) Authentic Assessment: The kind of assessment of student learning that measures 
student learning outcomes that are important (e.g., scientific principles, science process 
skills) rather than those that are easy to measure (e.g., facts and vocabulary); assessment 
is done in a way that truly demonstrates the learning that is valued (for example, having 
students actually use a thermometer to measure the temperature of the water in the 
beaker, rather than look at a picture of a thermometer in a beaker); authentic 
assessment is also done in the course of instruction, for the purpose of monitoring 
student understanding so activities can be adjusted ur new ones designed, rather than 
solely for the purpose of a final indication of whether or not learning has occurred. 

Learning Outcomes: see 2(a) 



Component 3: Teacher Development Program Activities 

(a) Adult Learners: Activities that are sensitive to the needs of adult learners have 
many attributes identified in the literature. These include: (1) a focus on growth rather 
than defects (e.g., activities model trying out new skills, getting feedback, and change; 
growth is rewarded; failure is expected and learned from); (2) activities are relevant and 
practical; (3) activities focus on individual interests and needs; (4) activities respond to 
the concerns or questions participants are asking; (5) activities link participants to and/or 
build in resources to support their learning (e.g., networking and mentoring). 

(c) Construct Knowledge: People don't learn by memorizing, but rather by connecting 
new information to what they already understand. Developing new concepts and 



The National Center for Improving Science Education 



34 



TDP-9 



strategies involves first being conscious of how one thinks things work (e.g., in the 
natural world or in the learning or teaching of young people), then fitting in the new 
ideas. Such "construction" of knowledge requires actual "hands-on" experiences and then 
opportunities to discuss, challenge each others 1 beliefs, and try out new ideas. The more 
that old and new ideas are discrepant, the more experiences and opportunities for 
discussion it takes for new ideas to replace or enhance old ones. 

(e) Immerse: Actually get involved in, e.g., teachers actually do science activities whose 
outcomes are the generation of scientific knowledge (mirroring the kinds of activities the 
lab conducts). 

(f) "Real World" Science: Related to applications of science in today's world: relevant, 
observable, applicable. 

(g) Learn Cooperatively: Work together in learning teams where tasks of learning are 
shared; includes discussions of observations to develop a shared view of what is 
happening; opportunities to develop and present common report or product. 

Component 5: Follow-Up 

Follow-Up: Structured, planned actions that take place as part of the program design, 
following the first major program or event; for the purpose of continuing and enhancing 
the goals of that program. May include continuing communication, workshops, classroom 
support, information, mentoring, equipment loan, or extension of the activity to a larger 
unit. 

Component 6: Teacher Leadership 

Leadership: Having the capability and skills to act as an agent for change. Actions 
include communication with others about their experiences, presentations for 
professional organizations, being active in professional organizations, working with 
authorities to facilitate change, working with policy makers, serving as master or mentor 
teacher, and working across grade levels and/or schools to implement change. 

Component 7: Program Evaluation 

(b) Formative and Summative Evaluation: Formative evaluation is done for the 
purpose of program improvement; it often focuses on the programs' activities and the 
reaction of participants to those activities. It is best done while the program is in 
progress, so that changes can be identified, made, and continually evaluated. Summative 
evaluation is done for the purpose of making judgments about the worth of the program. 
It often focuses on the over-time impact the program has on participants, and is often 
done after the program is completed or has been in place long enough to have stabilized. 
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Descriptive and Context Information 



Core Competencies: The set of basic areas of expertise and facilities upon which a 
specific laboratory builds its fundamental business (e.g., high energy physics, accelerators, 
materials science, high performance computing, etc.). 

Integration Activities: Activities that integrate all required research data and/or 
programs, and other laboratory expertise into a single solution for a particular problem 
(e.g., the combination of systems studies, materials research, engineering, computer 
simulations, and field testing into a solution to the problem of transporting of hazardous 
materials). 

Product Realization: The transfer of technology from the laboratories to the private 
sector, resulting in specific, marketable products. 
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Dear Reader: 

The purpose of this comment sheet is to find out how this booklet is being used, who is us- 
ing it, and how it might be improved. Please take a moment to fill it out. Thank you for your 
cooperation. 
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4. What do you find most valuable about this booklet? 



5. Are there specfic ideas, descriptions* or examples in this booklet that are not clear? If so, 
please give the page number and a brief description. 



6. Is there too little or too much information? If one or the other, please give the page num- 
ber and briefly describe what should be omitted or added. 
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